Communication device and related methods for automatically connecting to a captioning communication service to receive text captions following an interruption during a call

ABSTRACT

Apparatuses and methods are disclosed for automatically connecting to a relay service and establish a captioning communication session during a call between a hearing-impaired user and a far-end user. The apparatus includes communication elements configured to receive captions from a relay service, and a processor configured to automatically connect to the relay service and establish a captioning communication session during a call responsive to a determination that a connection to the relay service failed during a time in which the hearing-impaired user desired captions to be enabled. A method includes determining captions are desired by the hearing-impaired user at a first time to be enabled for display to the communication device during a call, detecting a failed connection to the relay service at a second time, and automatically connecting with the relay service and establishing a new captioning communication session at a third time during the call.

FIELD

The application relates generally to telecommunications and moreparticularly to communicating with a relay service for assistinghearing-impaired users in communicating with others. In addition, thedisclosure relates to automatically reconnecting to establish acaptioning communication session with the relay service following anunwanted interruption to the captioning communication session.

BACKGROUND

Hearing-impaired individuals may benefit from communication systems anddevices configured to provide assistance in order to communicate withother individuals over a communication network. For example, relayservices have been established to provide assistive services (e.g., textcaptions) to the hearing-impaired user communicating with acommunication device (e.g., caption phone, caption enabled device, etc.)that is specifically configured to communicate with the relay service.

In particular, a relay service may be a telecommunication intermediaryservice, which is intended to permit a deaf or a hearing-impaired personto utilize a normal telephone network. The relay service may include anoperator, referred to as a “call assistant,” who serves as a humanintermediary between the hearing-impaired user and a far-end user.During a captioning communication session, the call assistant may listento the audio signal of a far-end user and “revoice” the words of thefar-end user to a speech recognition computer program tuned to the voiceof the call assistant. Text captions (also referred to as “captions”)may be generated by the speech recognition computer as a transcriptionof the audio signal of the far-end user, and then transmitted to thecommunication device being used by the hearing-impaired user. Thecommunication device may then display the text captions while thehearing-impaired user carries on a normal conversation with the far-enduser. The text captions may allow the hearing-impaired user tosupplement the voice received from the far-end and confirm his or herunderstanding of the words spoken by the far-end user.

In some situations, the text captions may be transmitted to the user'scommunication device over a separate network than the voice signal fromthe far end. During a captioning communication session, there may be aproblem with the network that may cause the captioning communicationsession between the hearing-impaired user's communication device and therelay service to terminate. As a result, the hearing-impaired user'scommunication device may no longer receive and display the textcaptions. The termination of the connection may not have been initiatedor desired by the user, and the user may lose the benefit of the textcaptions. Conventionally, the user may need to press a caption button tostart another captioning communication session with the relay service toagain start to receive the text captions.

BRIEF SUMMARY

Embodiments of the disclosure include an apparatus associated with ahearing-impaired user. The apparatus comprises communication elementsand a processor operably coupled with the communication elements. Thecommunication elements are configured to receive audio signals from afar end communication device, and receive captions corresponding to theaudio signals from a relay service. The processor is configured toautomatically connect to the relay service and establish a captioningcommunication session during a call responsive to a determination that aconnection to the relay service failed during a time in which thehearing-impaired user desired captions to be enabled.

Also disclosed is a method of establishing a captioning communicationsession with a relay service for assisting a hearing-impaired user of acommunication device. The method comprises determining that captions aredesired by the hearing-impaired user at a first time to be enabled fordisplay to the communication device during a call, detecting a failedconnection to the relay service at a second time, and automaticallyconnecting with the relay service and establishing a new captioningcommunication session between the relay service and the communicationdevice at a third time during the call.

Also disclosed is an apparatus associated with a hearing-impaired user.The apparatus comprises an electronic display, communication elements,and a processor operably coupled with the electronic display and thecommunication elements. The communication elements are configured toreceive audio signals from a far end communication device, and receivecaptions corresponding to the audio signals from a relay service. Theprocessor is configured to establish a captioning communication sessionwith the relay service during a call with the far end communicationdevice, transmit the audio signals from the far end communication deviceto the relay service, receive text captions from the relay servicecorresponding to the audio signals from the far end communicationdevice, display the text captions on the electronic display, terminatethe captioning communication session responsive to a user input througha user interface of the apparatus, and automatically connect to therelay service and establish a new captioning communication sessionduring the call responsive to a determination that the captioningcommunication session is terminated in a manner that is different than auser input through the user interface.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates a communication system configured to facilitate acall between a hearing-impaired user and a far-end user.

FIG. 2 is a simplified schematic block diagram of a communication deviceassociated with a hearing-impaired user according to an embodiment ofthe disclosure.

FIG. 3 is a simplified block diagram of the software modules executed bythe processor of the communication device according to an embodiment ofthe disclosure.

FIG. 4 is a flowchart illustrating a method for operating acommunication device to automatically reconnect with a relay serviceaccording to an embodiment of the disclosure.

FIG. 5 is a state diagram for a state machine for the main moduleaccording to an embodiment of the disclosure.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and in which isillustrated specific embodiments in which the disclosure may bepracticed. These embodiments are described in sufficient detail toenable those of ordinary skill in the art to practice the disclosure. Itshould be understood, however, that the detailed description and thespecific examples, while indicating examples of embodiments of thedisclosure, are given by way of illustration only and not by way oflimitation. From this disclosure, various substitutions, modifications,additions, rearrangements, or combinations thereof within the scope ofthe disclosure may be made and will become apparent to those of ordinaryskill in the art.

In accordance with common practice, the various features illustrated inthe drawings may not be drawn to scale. The illustrations presentedherein are not meant to be actual views of any particular apparatus(e.g., device, system, etc.) or method, but are merely idealizedrepresentations that are employed to describe various embodiments of thedisclosure. Accordingly, the dimensions of the various features may bearbitrarily expanded or reduced for clarity. In addition, some of thedrawings may be simplified for clarity. Thus, the drawings may notdepict all of the components of a given apparatus (e.g., device) or alloperations of a particular method. In addition, like reference numeralsmay be used to denote like features throughout the specification andfigures.

Information and signals described herein may be represented using any ofa variety of different technologies and techniques. For example, data,instructions, commands, information, signals, bits, symbols, and chipsthat may be referenced throughout the description may be represented byvoltages, currents, electromagnetic waves, magnetic fields or particles,optical fields or particles, or any combination thereof. Some drawingsmay illustrate signals as a single signal for clarity of presentationand description. It should be understood by a person of ordinary skillin the art that the signal may represent a bus of signals, wherein thebus may have a variety of bit widths and the disclosure may beimplemented on any number of data signals including a single datasignal.

The various illustrative logical blocks, modules, circuits, andalgorithm acts described in connection with embodiments disclosed hereinmay be implemented or performed with a general-purpose processor, aspecial-purpose processor, a Digital Signal Processor (DSP), anApplication Specific Integrated Circuit (ASIC), a Field ProgrammableGate Array (FPGA) or other programmable logic device, discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein.

A processor herein may be any processor, controller, microcontroller, orstate machine suitable for carrying out processes of the disclosure. Aprocessor may also be implemented as a combination of computing devices,such as a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. When configured according toembodiments of the disclosure, a special-purpose computer improves thefunction of a computer because, absent the disclosure, the computerwould not be able to carry out the processes of the disclosure. Thedisclosure also provides meaningful limitations in one or moreparticular technical environments that go beyond an abstract idea. Forexample, embodiments of the disclosure provide improvements in thetechnical field of telecommunications, particularly in atelecommunication system including a relay service for providing textcaptions to a caption-enabled communication device to assisthearing-impaired users. Embodiments include features that improve thefunctionality of the communication device such that new communicationdevice and method for establishing captioning communication sessions aredescribed. As a result, the interaction of the communication device withother systems (e.g., the relay service) may be improved in addition toan improved user experience.

In addition, it is noted that the embodiments may be described in termsof a process that is depicted as a flowchart, a flow diagram, astructure diagram, or a block diagram. Although a flowchart may describeoperational acts as a sequential process, many of these acts can beperformed in another sequence, in parallel, or substantiallyconcurrently. In addition, the order of the acts may be re-arranged. Aprocess may correspond to a method, a function, a procedure, asubroutine, a subprogram, interfacing with an operating system, etc.Furthermore, the methods disclosed herein may be implemented inhardware, software, or both. If implemented in software, the functionsmay be stored or transmitted as one or more instructions (e.g., softwarecode) on a computer-readable medium. Computer-readable media includesboth computer storage media and communication media including any mediumthat facilitates transfer of a computer program from one place toanother.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth does not limit thequantity or order of those elements, unless such limitation isexplicitly stated. Rather, these designations may be used herein as aconvenient method of distinguishing between two or more elements orinstances of an element. Thus, a reference to first and second elementsdoes not mean that only two elements may be employed there or that thefirst element must precede the second element in some manner. Also,unless stated otherwise a set of elements may comprise one or moreelements.

As used herein, a “hearing-impaired user” may refer to a person withdiminished hearing capabilities. Hearing-impaired users ofcaption-enabled communication device often have some level of hearingability that has usually diminished over a period of time such that theycan communicate by speaking, but that they often struggle in hearingand/or understanding the far-end user.

The term “call” as used herein refers to the communication sessionbetween the hearing-impaired user's communication device and the far-enduser's communication device. The call may pass audio signals between thetwo parties. The term call is used in order to be more easilydistinguishable from the captioning communication session. At times, thecall may be referred to as incoming or outgoing from the perspective ofthe hearing-impaired user's communication device. Incoming and outgoingcalls may refer to the period of time prior to when the call is“answered” by the other party to begin the communication of the audiosignals therebetween.

The term “captioning communication session” as used herein refers to thecommunication session between the hearing-impaired user's communicationdevice and the relay service. The captioning communication session maypass text captions from the relay service to the hearing-impaired user'scommunication device. In some embodiments, the captioning communicationsession may also include the hearing-impaired user's communicationdevice transmitting the far-end user's audio signal to the relay serviceto generate the text captions.

FIG. 1 illustrates a communication system 100 configured to facilitatean assisted call between a hearing-impaired user 102 and a far-end user104. The communication system 100 may include a first communicationdevice 110, a second communication device 120, and a relay service 130.The first communication device 110 and the second communication device120 may be coupled together to facilitate communication therebetween viaa first network 140. The first communication device 110 and the relayservice 130 may be coupled together to facilitate communicationtherebetween via a second network 150. For example only, the firstnetwork 140 and the second network 150 may each be implemented accordingto the standards and bandwidth requirements of a communication network(e.g., Public Switch Telephone Network (PSTN), cellular network, VoiceOver Internet Protocol (VOIP) networks, etc.). The use of the terms“network” or “communication network” as used herein contemplatesnetworks that are compatible and configured to provide communicationsusing analog and/or digital standards unless specifically statedotherwise. In some embodiments, the first network 140 and the secondnetwork 150 may be the same network (e.g., both connections may beInternet-based connections). Thus, discussion of the first network 140and the second network 150 separately may be for convenience ofdiscussing a particular connection between two or more devices. Ofcourse, in some embodiments, the first network 140 and the secondnetwork 150 may be different networks. For example, the firstcommunication device 110 and the second communication device 120 maycommunicate via a PSTN network connection, while the first communicationdevice 110 and the second communication device 120 may communicate viaan internet connection. Other variations and combinations of networksare also contemplated.

The first communication device 110 may include a device that isconfigured to assist the hearing-impaired user 102 in communicating withanother individual (e.g., far-end user 104). In some embodiments, thefirst communication device 110 may include a caption-enabledcommunication device configured to receive and display text captions ofat least a portion of the conversation. Thus, the hearing-impaired user102 may be able to read the text captions of the words spoken by thefar-end user 104 to supplement the audio signal received by the firstcommunication device 110. As a result, the hearing-impaired user 102 mayhave an improved experience in understanding the conversation. Such anembodiment may be useful for people whose hearing has been damaged ordecreased over time (e.g., the elderly); such that they can still speakbut have diminished hearing that makes it difficult to communicate. Insome embodiments, the first communication device 110 may also beconfigured to receive and display video on an electronic display on thefirst communication device 110.

The second communication device 120 may comprise a conventional voicetelephone (e.g., landline phone, cellular phone, smart phone, VoIPphone, etc.). As such, the far-end user 104 may interact in aconventional manner with the second communication device 120. In someembodiments, the second communication device 120 may be configuredsimilarly as the first communication device (e.g., caption-enabledcommunication device). As a result, the second communication device 120may likewise be operated by a hearing-impaired user. Thus, althoughfacilitating communication between the hearing-impaired user 102 and thefar-end user 104 is shown in FIG. 1 to imply that the far-end user 104is a hearing-capable user, such a situation is shown only as an example.Other embodiments include both the first communication device 110 andthe second communication device 120 coupled to the relay service 130 tofacilitate the captioning services for each respective hearing-impaireduser. In such a situation, each communication device 110, 120 may haveits own communication session with the relay service 130.

The relay service 130 may be configured to provide interpretive services(e.g., captioning) to the hearing-impaired user 102. More specifically,a human “call assistant” within relay service 130 may be employed tofacilitate an assisted call between a hearing-impaired user 102 and afar-end user 104. As discussed above, in some embodiments the relayservice 130 may be configured to provide text captions of at least aportion of the conversation. In such an embodiment, the call assistantmay listen to the voice signal received and re-voice the portion of theconversation into a microphone so that voice recognition software maygenerate the text captions that are transmitted to the firstcommunication device 110. Thus, the relay service 130 may include one ormore of an internet protocol captioned telephone service (IPCTS),captioned telephone service (CTS), or other telecommunications relayservices (TRS).

FIG. 1 shows a configuration where the first communication device 110acts as a router for the voice signal from the second communicationdevice 120 to the relay service. In such an embodiment, the voice signalof the far-end user 104 may be transmitted from the second communicationdevice 120 to the first communication device 110. The voice signal ofthe far-end user 104 may then be transmitted from the firstcommunication device 110 to the relay service 130 for the text captionsto be generated in a text captioning embodiment. The text captions maythen be transmitted from the relay service 130 to the firstcommunication device 110 to be displayed as text captions for thehearing-impaired user to read during the conversation. The callassistant may also monitor the text captions that are generated andtransmitted to the first communication device 110 to identify any errorsthat may have been generated by the voice recognition software. The callassistant may correct such errors, such as described in U.S. Pat. No.8,379,801, issued Feb. 19, 2013, entitled “Methods and Systems Relatedto Text Caption Error Correction,” the disclosure of which isincorporated herein in its entirety by this reference. In someembodiments the relay service 130 may be configured to receive the voicesignal from the second communication device 120 and route the voicesignal to the first communication device 110.

In addition, although FIG. 1 shows only two communication devices 110,120, the communication system 100 may include more communicationdevices. It is contemplated that the communication system 100 mayfacilitate communication between any number and combinations ofhearing-impaired users and far-end users. For example, in someembodiments two or more communication devices may be connected forfacilitating communication between a hearing-impaired user and otherhearing-impaired users and/or far-end users.

As discussed above, during a captioning communication session, theconnection between the first communication device 110 and the relayservice 130 may be interrupted. In some situations, the hearing-impaireduser may have desired to disconnect from the relay service 130. Forexample, the first communication device 110 may have a “caption button”that the user may select to enable or disable captioning. The captionbutton may be a mechanical input on the first communication device 110itself and/or an input on a graphical user interface that the user mayselect. The term “button” is used herein for convenience in describingany input element into the communication device for enabling captioning.Thus, the term “button” may include other types of input elements thatmay be selected through methods other than by a pressing action. Forexample, the user may enable captioning by a sliding motion on an iconof a touch screen interface or other similar methods.

Enabling captioning may cause a captioning communication session to beestablished between the first communication device 110 and the relayservice 130 for the first communication device 110 to transmit the audiosignal from the far end user to the relay service 130, and to receivethe text captions from the relay service 130 for display on the firstcommunication device 110. Disabling captioning may cause the captioningcommunication session between the first communication device 110 and therelay service 130 to be terminated. This termination may be independentof the call between the first communication device 110 and the secondcommunication device 120. In other words, captioning may be enabledand/or disabled during a call as desired by the hearing-impaired user.At the relay service side, when captioning is disabled, the callassistant may simply move on to the next call assigned to the callassistant by the relay service 130.

In some situations, the hearing-impaired user may not have desired todisable captioning during a call. For example, some contributing factorsmay be local issues for the hearing-impaired user, such as a networkcable to the first communication device 110 being unplugged, a routerbeing disabled, a spotty WiFi connection, etc. Some contributing factorsmay be relay service side issues, such as a server problem, terminationby a call assistant, etc. Other contributing factors may be issues alongthe path between the first communication device 110 and the relayservice 130, such as a network outage. In any event, thehearing-impaired user may have desired captioning, but the captioningmay have been terminated during a call against the wishes of thehearing-impaired user (e.g., the hearing-impaired user did not selectthe caption button to disable captioning).

The first communication device 110 may be configured to automaticallyreconnect to the relay service 130 and establish a captioningcommunication session in response to detecting that the priorcommunication session was terminated during the call without the userselecting the caption button. In some embodiments, the new communicationsession may be with a different call assistant, whereas in someembodiments, the new communication session may be with the same callassistant from the prior communication session.

FIG. 2 is a simplified schematic block diagram of a communication device200 associated with a hearing-impaired user according to an embodimentof the disclosure. For example, the communication device 200 may be thefirst communication device 110 of FIG. 1. In particular, thecommunication device 200 may be configured to establish calls with othercommunication devices and captioning communication sessions with a relayservice configured to assist the hearing-impaired user. Thecommunication device 200 may be a caption enabled communication device,which may be implemented as a standalone device (e.g., a caption phone),or as implemented on another device (e.g., tablet computer, laptopcomputer, smart phone, etc.).

The communication device 200 may include a processor 210 operablycoupled with an electronic display 220, communication elements 230, amemory device 240, input devices 250, and a speaker 260. In someembodiments, the communication device 200 may include a camera for alsoparticipating in a video communication session. The processor 210 maycoordinate the communication between the various devices as well asexecute instructions stored in computer-readable media of the memorydevice 240. The processor 210 may be configured to execute a widevariety of operating systems and applications including the computinginstructions. The memory device 240 may be used to hold computinginstructions, data, and other information for performing a wide varietyof tasks including performing embodiments disclosed herein. By way ofexample and not limitation, the memory device 240 may includeSynchronous Random Access Memory (SRAM), Dynamic RAM (DRAM), Read-OnlyMemory (ROM), Flash memory, and the like. The memory device 240 mayinclude volatile and non-volatile memory storage for the communicationdevice 200.

The communication elements 230 may be configured to communicate withother devices or communication networks, including other communicationdevices and the relay service. As non-limiting examples, thecommunication elements 230 may include elements for communicating onwired and wireless communication media, such as for example, serialports, parallel ports, Ethernet connections, universal serial bus (USB)connections IEEE 1394 (“firewire”) connections, Bluetooth wirelessconnections, 802.1 a/b/g/n type wireless connections, and other suitablecommunication interfaces and protocols. The input devices 250 mayinclude a numeric keypad, a keyboard, a touchscreen, a remote control, amouse, buttons, other input devices, or combinations thereof.

FIG. 3 is a simplified block diagram 300 of the software modulesexecuted by the processor of the communication device according to anembodiment of the disclosure. The modules include a main module 310, afar end communication module 320, a relay service session module 330, arelay service communication module 340, and a user interface (UI) module350. The main module 310 may be configured to coordinate communicationwith each of the other modules 320, 330, 340, 350.

The far end communication module 320 may be configured to receive theaudio data from the far end communication device and transmit the localaudio data to the far end communication device. The far endcommunication module 320 may provide the interface between thecommunication line (e.g., PSTN, VoIP) and the main module 310 tocommunicate the audio data between the communication device and the farend communication device.

The relay service session module 330 may be configured to establish acaptioning communication session with the relay service when captionsare desired by the user. Establishing the captioning communicationsession may be responsive to the user selecting a captioning button. Insome embodiments, establishing the captioning communication session maybe responsive to the call itself such that the captions are enabled bydefault. For embodiments that use session initiation protocol (SIP), therelay service session module 330 may be configured as a SIP stack. Otherprotocols are also contemplated.

The relay service communication module 340 may be configured to transmitand receive data between the communication device and the relay service.For example, the relay service communication module 340 may be thetransport layer for communicating through the captioning communicationsession established by the relay service session module 330. The relayservice communication module 340 may transmit the far end audio signal(e.g., in audio packets) to the relay service and receive the textcaptions of the audio signal in return. For embodiments that useReal-time Transport Protocol (RTP), the relay service communicationmodule 340 may be configured as an RTP stack. Other protocols are alsocontemplated.

The user interface module 350 may be configured to communicate with theuser interface provided to the user. The user interface may includeinput and output devices that may include mechanical interfaces (e.g.,mechanical buttons, switches, etc.) and/or graphical interfaces on theelectronic display screen. For example, the user interface module 350may receive inputs from the user (e.g., for placing/terminating calls,enabling/disabling captions, updating contacts, viewing callinformation, changing settings, etc.).

In addition to coordinating the communication with each of the othermodules 320, 330, 340, 350, the main module 310 may determine whether ornot a termination of the captioning communication session with the relayservice was intentional, and to automatically reconnect with the relayservice during the same call responsive to a determination that that thetermination was unintentional. For example, if the main module 310determines that the termination of the captioning communication sessionwas unintentional, the main module 310 may instruct the relay servicesession module 330 to establish another communication session with therelay service.

In some embodiments, the main module 310 may determine whether thetermination was intentional by determining which module 320, 330, 340,350 informed the main module 310 that the captioning communicationsession is to be terminated and/or has been terminated. For example, ifthe main module 310 receives a termination request from the userinterface module 350, the main module 310 may determine that thetermination request is responsive to a user input (e.g., selecting thecaption button to disable captions, etc.), which may indicate that theuser intentionally initiated the termination request.

If the main module 310 receives a message from the relay service sessionmodule 330 indicating that the captioning communication session has beenterminated, the main module 310 may determine that the termination wasnot responsive to a user input, which may indicate that the user did notintentionally initiated the termination request. As an example, therelay service session module 330 may periodically transmit a pingmessage to the relay service 130 and wait for an acknowledgement messagein return. If the acknowledgement message is not received by the relayservice session module 330 within a predetermined period of time (e.g.,30 seconds), the relay service session module 330 may send an errormessage to the main module that the captioning communication session hasbeen terminated.

If the main module 310 receives a message from the relay servicecommunication module 340 indicating that the captioning communicationsession has been terminated, the main module 310 may determine that thetermination was not responsive to a user input, which may indicate thatthe user did not intentionally initiate the termination request. Forexample, the relay service communication module 340 may detect an errorin the transmission of the audio signal to the relay service (e.g., whenthe audio packets are transmitted), and send an error message to themain module that the captioning communication session has beeninterrupted. In another situation, the call assistant may activelyterminate the captioning communication session at the relay service. Asa result, the relay service may send a termination request to thecommunication device. As a result, the relay service communicationmodule 340 send the termination request to the main module 310. With thetermination request not being initiated by the hearing-impaired user,the main module 310 may determine that the hearing-impaired user maystill desire captions. Therefore, the main module 310 may instruct therelay service session module 330 to establish another communicationsession with the relay service responsive to the call assistantterminating the captioning communication session.

FIG. 4 is a flowchart 400 illustrating a method for operating acommunication device to automatically reconnect with a relay serviceaccording to an embodiment of the disclosure. At operation 410, theprocessor may determine whether captioning has been enabled and, if so,establish a captioning communication session with the relay service atoperation 420. For example, in some embodiments, the user may select acaption button on the communication device to enable captioning. Theuser may select the caption button during a call, while a call isincoming (e.g., before the call is answered), prior to making anoutbound call, etc. In some embodiments, captioning may be enabled bydefault. In other words, when a captioning communication session isestablished with a far end communication device, captioning may beenabled by default such that a captioning communication session with therelay service may be established without an affirmative selection of acaption button. Thus, a captioning communication session with the relayservice may be automatically established responsive to the call itself,such as while waiting for an answer or when the other party answers thecall.

At operation 430, text captions may be received and displayed by thecommunication device from the relay service during the call. During thistime, the communication device may also be routing the audio signal fromthe far end to the relay service to generate the text captions. At anytime during the call, the user may decide to disable captioning. Forexample, the user may select the caption button. Thus, the processor maydetermine whether the hearing-impaired user has affirmatively disabledthe captioning at operation 440. If so, the captioning communicationsession between the communication device and the relay service may beterminated at operation 450. The call between the communication deviceand the far end communication device may continue and the processor mayawait a user input to again enable captioning if desired by thehearing-impaired user. In some embodiments, disabling the captioning maybe detected based on the call between the communication device and thefar end communication device ending (e.g., one party hanging up).

At operation 460, the processor may detect interruption with thecaptioning communication session with the relay service. Thisinterruption may not have been through the normal methods for the userto indicate a desire to disable captioning. For example, the captioningcommunication session may be terminated by a problem with the localnetwork connection of the communication device (e.g., spotty WiFiconnection, disabled router, unplugged cable, etc.) or external networkproblems or even issues with the relay service (e.g., server error, callassistant initiated disconnect, etc.).

If the processor does not detect such an interruption in the captioningcommunication session with the relay service, the communication devicemay continue to receive captions for display to the hearing-impaireduser during the call at operation 430. If, however, the processor doesdetect such an interruption, the communication device may return tooperation 420 to establish a captioning communication session with therelay service if the connection to the relay service becomes availableduring the present call. If the connection to the relay service is notavailable, the communication device may continue to attempt to establishanother communication session until the present call is terminated. Insome embodiments, the communication device may attempt to establishanother communication session according to a predetermined interval(e.g., every 5 seconds) until either the new communication session isestablished or the present call is terminated. In some embodiments, thecommunication device may attempt to establish another communicationsession according to staggered intervals. For example, a first attemptmay occur after 2 seconds, a second attempt may occur after another 4seconds, a third attempt may occur after another 8 seconds, a fourthattempt may occur after another 16 seconds, etc. At some point, thestaggered interval for attempts may be increased up to a maximuminterval (e.g., every 30 seconds) until the call ends.

Thus, the communication device may be automatically reconnected to therelay service during the same call to establish a new communicationsession and receive captions when the connection to the relay service isavailable. As a result, the burden on the hearing-impaired user tocontinue trying to begin captioning again may be reduced, and the amountof time that the captions are unintentionally disabled may be reduced.If the new communication session is established, the relay service mayassign the new communication session to a new call assistant who is thenext available in line. This may be the case as the prior call assistantmay have already been assigned a new call from a different communicationdevice. In some embodiments, the new communication session may beassigned to the same call assistant as the prior communication session.In some situations, this may be purely coincidental, while in othersituations it may be desirable for the system to attempt to ensure thesame call assistant is assigned to the new communication session forconsistency, as the call assistant is already familiar with the speakingpatterns of the far-end user as it is part of the same call as before.For example, the communication device may have a call assistantidentifier during the captioning communication session so that when therequest for reconnection occurs, the communication device may send thecall assistant identifier to the relay service with the request. In someembodiments, the relay service may store a record of the call assistantsand communication devices used during a specific call such that therelay service may recognize a request for reconnection for the samecall. As a result, the new session may be reassigned to the same callassistant, if available. In such an embodiment, there may be a waitingperiod (e.g., 1 minute) that may elapse before the call assistant isassigned to the next available call while waiting for reconnection.

FIG. 5 is a state diagram for a state machine for the main module 310 ofFIG. 3 according to an embodiment of the disclosure. The main module 310may begin in a disconnected state 510 during which time the main module310 may await instructions to enable captioning. In some embodiments,the instructions may be responsive to the user selecting a captionbutton to enable the captions during a call. In some embodiments, theinstructions may be responsive to a “default on” setting in which thecall itself enables captions and requests a captioning communicationsession with the relay service.

The main module 310 may then enter into a session opening state 520 andinstruct the relay service session module 330 to transmit a sessioninvitation to the relay service for requesting a captioningcommunication session with the relay service. The relay service sessionmodule 330 may receive an acknowledgement from a hold server while therelay service waits for an available call assistant, and the main module310 may enter into an on hold state 530.

The relay service session module 330 may then receive another messageindicating that the call has been accepted by the relay service andassigned to a call assistant. The message may also include the port IDand IP address for the main module 310 to instruct the relay servicecommunication module 340 to transmit the audio signal and receive thetext captions. The main module 310 may then enter a connected state 540.

At some point during the connected state 540, the hearing-impaired usermay desire to terminate the connection to the relay service. Forexample, the hearing-impaired user may select the captioning button todisable captioning during a call. In some situations, the call itselfmay simply end. In either situation, the main module 310 may enter intoa session closing state 550. In the session closing state 550, thecommunication device may send a message to the relay service (e.g.,through the relay service communication module 340) to inform the relayservice that the captioning communication session will be ending andthat no more audio signals will be sent. The main module 310 may thenreturn to the disconnected state 510. In some embodiments, the mainmodule 310 may return to the disconnected state 510 responsive toreceiving an acknowledgement from the relay service that the captioncommunication is over. In some embodiments, the main module 310 mayreturn to the disconnected state 510 responsive to a predeterminedtimeout period (e.g., 5 seconds) expiring if the acknowledgement fromthe relay service is not received. During this predetermined timeoutperiod, the communication device may continue to receive text captionseven though additional audio has not been sent in case the relay servicehas not yet sent the text captions for the audio signal that was sentprior to entering the session closing state 550 and is still inprogress.

At any time during the session opening state 520 or the on hold state530, the call itself may be terminated. In such a situation, the mainmodule 310 may simply return to the disconnected state 510 as acaptioning communication session had not yet been established. However,during the connected state 540, termination of the call itself may stillcause the main module 310 to enter the session closing state 550 so thatthe established captioning communication session may be closed and thein progress text captions may be received. In some embodiments,terminating the call during the session opening state 520 or the on holdstate 530 may cause the main module 310 to enter the session closingstate 550 to shut down the connection in an orderly fashion.

In addition, at any time during the session opening state 520, the onhold state 530, or the connected state 540, the captioning communicationsession may be terminated due to events that are unintended by thehearing-impaired user. As discussed above, such events may includespotty WiFi, disconnected cables, disconnected router, network issuesalong the path to the relay service, server problems at the relayservice, termination by the call assistant, and other issues that maycause the captioning communication session to be terminated even thoughthe hearing-impaired user may still desire captioning during the call.The main module 310 may detect that one of these situations has occurredby recognizing that the termination message was received by a moduleother than the user interface module. For example, error messages may begenerated by the relay service session module 330 and/or the relayservice communication module 340 and sent to the main module 310. Insome embodiments, the main module 310 may detect that one of thesesituations has occurred by recognizing that the termination requestoccurred during a state 520, 530, 540 in which such a terminationrequest is not expected.

In such situations, the main module 310 may return to the disconnectedstate 510 and automatically enter the session opening state 520 toinstruct the relay service session module 330 to send another sessionrequest to the relay service. If the connection to the relay service isstill not available, the main module 310 may continue to instruct therelay service session module 330 to send additional session requests tothe relay service until a new communication session is established oruntil the present call ends. Such session requests may occur at regularintervals and/or staggered intervals as desired.

The examples discussed above generally discuss the situation in whichthe captions are enabled during a call (whether by default or userselection), and where the established captioning communication sessionis subsequently terminated. Some embodiments may also include thesituations in which the user has indicated that captions are desired,but that the captioning communication session was never establishedbecause the connection to the relay service was unavailable at thebeginning. For example, the hearing-impaired user may select the captionbutton to enable captions at a time that the network to connect to therelay service was unavailable. Once the hearing-impaired user has madethis indication, the main module 310 may continue to instruct the relayservice session module 330 to attempt to automatically establish thecaptioning communication session if the connection to the relay servicebecomes available during the call. Similarly, a communication devicethat has captions set to be on by default may not be able to establishthe captioning communication session at the beginning of the call. As aresult, the main module may continue to instruct the relay servicesession module 330 to attempt to automatically establish the captioningcommunication session if the connection to the relay service becomesavailable during the call.

While certain illustrative embodiments have been described in connectionwith the figures, those of ordinary skill in the art will recognize andappreciate that embodiments encompassed by the disclosure are notlimited to those embodiments explicitly shown and described herein.Rather, many additions, deletions, and modifications to the embodimentsdescribed herein may be made without departing from the scope ofembodiments encompassed by the disclosure, such as those hereinafterclaimed, including legal equivalents. In addition, features from onedisclosed embodiment may be combined with features of another disclosedembodiment while still being encompassed within the scope of embodimentsencompassed by the disclosure as contemplated by the inventors.

What is claimed is:
 1. An apparatus associated with a hearing-impaireduser, the apparatus comprising: communication elements configured toreceive audio signals from a far end communication device, and receivecaptions corresponding to the audio signals from a captioningcommunication service that provides captioning assistance forhearing-impaired users; and a processor operably coupled with thecommunication elements, the processor configured to: connect with thefar-end communication device through a first connection to establish avoice communication session between the apparatus and the far endcommunication device; and automatically connect with the captioningcommunication service through a second connection to establish acaptioning communication session during the voice communication sessionresponsive to a determination that an attempted connection to thecaptioning communication service failed during a time in which a captionselect option has been enabled by the hearing-impaired user, wherein theattempted automatic connection through the second connection is not madeduring a time in which the caption select option has been disabled bythe hearing-impaired user, and wherein the first connection with thefar-end communication device is maintained until being disconnected bythe hearing-impaired user in both situations in which the caption selectoption is either enabled or disabled.
 2. The apparatus of claim 1,wherein the processor is configured to automatically connect with thecaptioning communication service through the second connectionresponsive to an error message from a captioning communication servicesession module executed by the processor of the apparatus that isconfigured to establish communication sessions between the apparatus andthe captioning communication service.
 3. The apparatus of claim 2,wherein the captioning communication service session module includes asession initiated protocol (SIP) stack.
 4. The apparatus of claim 2,wherein the error message includes an error message generated by thecaptioning communication service session module after expiration of atime out period following a ping to the captioning communication serviceduring the voice communication session.
 5. The apparatus of claim 1,wherein the processor is configured to automatically connect with thecaptioning communication service through the second connectionresponsive to a message from a captioning communication servicecommunication module executed by the processor of the apparatus that isconfigured to receive the text captions from the captioningcommunication service.
 6. The apparatus of claim 5, wherein the messageincludes an error message generated by the captioning communicationservice communication module responsive to detecting a failure oftransmission of audio signals from the apparatus to the captioningcommunication service.
 7. The apparatus of claim 5, wherein the messageincludes a termination message generated by the captioning communicationservice and received by the processor of the apparatus.
 8. An apparatusassociated with a hearing-impaired user, the apparatus comprising:communication elements configured to receive audio signals from a farend communication device, and receive captions corresponding to theaudio signals from a captioning communication service that providescaptioning assistance for hearing-impaired users; and a processoroperably coupled with the communication elements, the processorconfigured to: connect with the far-end communication device through afirst connection to establish a voice communication session between theapparatus and the far end communication device; and automaticallyconnect with the captioning communication service through a secondconnection to establish a captioning communication session during thevoice communication session responsive to a determination that anattempted connection to the captioning communication service failedduring a time in which a caption select option has been enabled by thehearing-impaired user, wherein the attempted automatic connectionthrough the second connection is not made during a time in which thecaption select option has been disabled by the hearing-impaired user,wherein the first connection with the far-end communication device ismaintained until being disconnected by the hearing-impaired user in bothsituations in which the caption select option is either enabled ordisabled, and wherein the processor is configured to automaticallyconnect with the captioning communication service through the secondconnection responsive to a termination message being received by amodule other than a user interface module, while a main module executedby the processor is in a state of the main module that is different thana session closing state of the main module.
 9. A method of establishinga captioning communication session with a captioning communicationservice for assisting a hearing-impaired user of a communication device,the method comprising: establishing a voice communication sessionbetween a captioning communication device and a far end communicationdevice through a first connection; determining that captions are desiredby the hearing-impaired user at a first time to be enabled for displayto the communication device during the voice communication sessionresponsive to a caption select option being enabled; detecting a failedconnection between the captioning communication device and thecaptioning communication service through a second connection at a secondtime during the voice communication session while the caption selectoption is still enabled; and automatically attempting to connect thecaptioning communication device and the captioning communication serviceuntil at intervals during the voice communication session without userinput until a new captioning communication session is establishedbetween the captioning communication service and the communicationdevice, the caption select option is disabled, or until the voicecommunication session is terminated, wherein the first connection ismaintained until the voice communication session is terminated; andestablishing the new captioning communication session between captioningcommunication service and the communication device at a third timeduring the voice communication session if the automatic attempt toconnect through the second connection is successful.
 10. The method ofclaim 9, wherein determining captions are desired by thehearing-impaired user at the first time to be enabled responsive to thecaption select option being enabled includes receiving an inputselection at the communication device by the hearing-impaired user. 11.The method of claim 9, wherein determining captions are desired by thehearing-impaired user at the first time to be enabled responsive to thecaption select option being enabled includes the communication devicedetermining captions to be enabled by default.
 12. The method of claim9, wherein detecting the failed connection between the captioningcommunication device and the captioning communication service throughthe second connection at the second time includes receiving an errormessage generated internally within the caption communication devicewithout a selection of the hearing-impaired user being received througha user interface of the communication device.
 13. The method of claim 9,wherein detecting the failed connection between the captioningcommunication device and the captioning communication service throughthe second connection at the second time occurs after the caption selectoption has already been enabled for the captioning communication deviceduring a prior captioning communication session during the same voicecommunication session.
 14. The method of claim 9, wherein automaticallyattempting to connect the captioning communication device and thecaptioning communication service through the second connection includessending a new session request from the captioning communication deviceto the captioning communication service at each interval.
 15. The methodof claim 14, wherein the intervals are regular intervals.
 16. The methodof claim 14, wherein the intervals are staggered intervals.
 17. Themethod of claim 13, wherein establishing the new captioningcommunication session includes the new captioning communication sessionbeing assigned to a different communication assistant at the captioningcommunication service than with the prior captioning communicationduring the same voice communication session.
 18. The method of claim 13,wherein establishing the new captioning communication session includesthe new captioning communication session being assigned to a samecommunication assistant at the captioning communication service as withthe prior captioning communication during the same voice communicationsession.
 19. The method of claim 9, wherein detecting a failedconnection between the captioning communication device and thecaptioning communication service through the second connection at thesecond time includes the captioning communication device receiving atermination request from a communication assistant assigned to a currentcaptioning communication session at the captioning communication servicewhile the voice communication session is still ongoing through the firstconnection.
 20. The method of claim 9, wherein the first time and thesecond time occur at least substantially simultaneously.
 21. Anapparatus associated with a hearing-impaired user, the apparatuscomprising: an electronic display; communication elements configured toreceive audio signals from a far end communication device, and receivecaptions corresponding to the audio signals from a captioningcommunication service; and a processor operably coupled with theelectronic display and the communication elements, the processorconfigured to: establish a voice communication session through a firstconnection with the far end communication device; establish a captioningcommunication session through a second connection with the captioningcommunication service during the voice communication session with thefar end communication device; transmit the audio signals received fromthe far end communication device during the voice communication sessionto the captioning communication service over the second connection;receive text captions from the captioning communication service throughthe second connection, the text captions corresponding to the audiosignals received from the far end communication device during the voicecommunication session; display the text captions on the electronicdisplay; enable the captioning communication session responsive to acaption select option being enabled based on either a user input througha user interface of the apparatus or through a caption default onsetting for the apparatus; disable the captioning communication sessionresponsive to the caption select option being disabled based on either auser input through a user interface of the apparatus or through acaption default on setting for the apparatus; and automatically connectto the captioning communication service through the second connectionand establish a new captioning communication session during the voicecommunication session after the captioning communication session hasbeen enabled and responsive to a determination that the captioningcommunication session is terminated in a manner that is different than auser input being received through the user interface, wherein the firstconnection is maintained while automatic reconnection occurs unless thecaption select option is disabled or until the voice communicationsession is terminated.
 22. The method of claim 9, wherein automaticallyconnecting the captioning communication device and the captioningcommunication service through the second connection and establishing thenew captioning communication session includes reconnecting thecaptioning communication device and the captioning communication servicethrough the second connection after the communication device and thecaptioning communication service had already been connected for aprevious captioning communication session during the same voicecommunication session between the communication device and a far endcommunication device.
 23. The method of claim 21, wherein the processorconfigured to automatically connect to the captioning communicationservice through the second connection by sending a plurality of sessionrequests from the captioning communication device to the captioningcommunication service at continuing intervals without user input whilethe voice communication session between the communication device and thefar end communication device is connected or until the new captioningcommunication session is established.